---
id: benchmarks-drop
title: DROP
sidebar_label: DROP
---

<head>
  <link rel="canonical" href="https://deepeval.com/docs/benchmarks-drop" />
</head>

**DROP (Discrete Reasoning Over Paragraphs)** is a benchmark designed to evaluate language models' advanced reasoning capabilities through complex question answering tasks. It encompasses over 9500 intricate challenges that demand numerical manipulations, multi-step reasoning, and the interpretation of text-based data. For more insights and access to the dataset, you can [read the original DROP paper here](https://arxiv.org/pdf/1903.00161v2.pdf).

:::info
`DROP` challenges models to process textual data, **perform numerical reasoning tasks** such as addition, subtraction, and counting, and also to **comprehend and analyze text** to extract or infer answers from paragraphs about **NFL and history**.
:::

## Arguments

There are **TWO** optional arguments when using the `DROP` benchmark:

- [Optional] `tasks`: a list of tasks (`DROPTask` enums), which specifies the subject areas for model evaluation. By default, this is set to all tasks. The list of `DROPTask` enums can be found [here](#drop-tasks).
- [Optional] `n_shots`: the number of examples for few-shot learning. This is **set to 5** by default and **cannot exceed 5**.

:::note
Notice unlike `BIGBenchHard`, there is no CoT prompting for the `DROP` benchmark.
:::

## Usage

The code below assesses a custom mistral_7b model ([click here](/guides/guides-using-custom-llms) to learn how to use ANY custom LLM) on `HISTORY_1002` and `NFL_649` in DROP using 3-shot prompting.

```python
from deepeval.benchmarks import DROP
from deepeval.benchmarks.tasks import DROPTask

# Define benchmark with specific tasks and shots
benchmark = DROP(
    tasks=[DROPTask.HISTORY_1002, DROPTask.NFL_649],
    n_shots=3
)

# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)
```

The `overall_score` for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on **exact matching**, is calculated by determining the proportion of questions for which the model produces the precise correct answer (e.g. '3' or ‘John Doe’) in relation to the total number of questions.

As a result, utilizing more few-shot prompts (`n_shots`) can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall score.

## DROP Tasks

The DROPTask enum classifies the diverse range of categories covered in the DROP benchmark.

```python
from deepeval.benchmarks.tasks import DROPTask

drop_tasks = [NFL_649]
```

Below is the comprehensive list of available tasks:

- `NFL_649`
- `HISTORY_1418`
- `HISTORY_75`
- `HISTORY_2785`
- `NFL_227`
- `NFL_2684`
- `HISTORY_1720`
- `NFL_1333`
- `HISTORY_221`
- `HISTORY_2090`
- `HISTORY_241`
- `HISTORY_2951`
- `HISTORY_3897`
- `HISTORY_1782`
- `HISTORY_4078`
- `NFL_692`
- `NFL_104`
- `NFL_899`
- `HISTORY_2641`
- `HISTORY_3628`
- `HISTORY_488`
- `NFL_46`
- `HISTORY_752`
- `HISTORY_1262`
- `HISTORY_4118`
- `HISTORY_1425`
- `HISTORY_460`
- `NFL_1962`
- `HISTORY_1308`
- `NFL_969`
- `NFL_317`
- `HISTORY_370`
- `HISTORY_1837`
- `HISTORY_2626`
- `NFL_987`
- `NFL_87`
- `NFL_2996`
- `NFL_2082`
- `HISTORY_23`
- `HISTORY_787`
- `HISTORY_405`
- `HISTORY_1401`
- `HISTORY_835`
- `HISTORY_565`
- `HISTORY_1998`
- `HISTORY_2176`
- `HISTORY_1196`
- `HISTORY_1237`
- `NFL_244`
- `HISTORY_3109`
- `HISTORY_1414`
- `HISTORY_2771`
- `HISTORY_3806`
- `NFL_1233`
- `NFL_802`
- `HISTORY_2270`
- `NFL_578`
- `HISTORY_1313`
- `NFL_1216`
- `NFL_256`
- `HISTORY_3356`
- `HISTORY_1859`
- `HISTORY_3103`
- `HISTORY_2991`
- `HISTORY_2060`
- `HISTORY_1408`
- `HISTORY_3042`
- `NFL_1873`
- `NFL_1476`
- `NFL_524`
- `HISTORY_1316`
- `HISTORY_1456`
- `HISTORY_104`
- `HISTORY_1275`
- `HISTORY_1069`
- `NFL_3270`
- `NFL_1222`
- `HISTORY_2704`
- `HISTORY_733`
- `NFL_1981`
- `NFL_592`
- `HISTORY_920`
- `HISTORY_951`
- `NFL_1136`
- `HISTORY_2642`
- `HISTORY_1065`
- `HISTORY_2976`
- `NFL_669`
- `HISTORY_2846`
- `NFL_1996`
- `HISTORY_2848`
- `NFL_3285`
- `HISTORY_2789`
- `HISTORY_3722`
- `HISTORY_514`
- `HISTORY_869`
- `HISTORY_2857`
- `HISTORY_3237`
- `NFL_563`
- `HISTORY_990`
- `HISTORY_2961`
- `NFL_3387`
- `HISTORY_124`
- `HISTORY_2898`
- `HISTORY_2925`
- `HISTORY_2788`
- `HISTORY_632`
- `HISTORY_2619`
- `HISTORY_3278`
- `NFL_749`
- `HISTORY_3726`
- `NFL_1096`
- `NFL_1207`
- `HISTORY_3079`
- `HISTORY_2939`
- `HISTORY_3581`
- `NFL_2777`
- `HISTORY_3873`
- `HISTORY_1731`
- `HISTORY_426`
- `NFL_1478`
- `HISTORY_3106`
- `NFL_1498`
- `NFL_3133`
- `HISTORY_3345`
- `NFL_503`
- `HISTORY_801`
- `NFL_2931`
- `NFL_2482`
- `HISTORY_1945`
- `NFL_2262`
- `HISTORY_3735`
- `HISTORY_1151`
- `NFL_2415`
- `HISTORY_607`
- `HISTORY_724`
- `HISTORY_1284`
- `HISTORY_494`
- `NFL_3571`
- `NFL_1307`
- `HISTORY_2847`
- `HISTORY_2650`
- `NFL_1586`
- `NFL_2478`
- `HISTORY_1276`
- `NFL_540`
- `NFL_894`
- `NFL_1492`
- `HISTORY_3265`
- `HISTORY_686`
- `HISTORY_2546`
- `NFL_2396`
- `HISTORY_2001`
- `HISTORY_1793`
- `HISTORY_2014`
- `HISTORY_2732`
- `HISTORY_2927`
- `NFL_1195`
- `HISTORY_1650`
- `NFL_2077`
- `HISTORY_3036`
- `HISTORY_495`
- `HISTORY_3048`
- `HISTORY_912`
- `HISTORY_936`
- `NFL_1329`
- `HISTORY_1928`
- `HISTORY_3303`
- `HISTORY_2199`
- `HISTORY_1169`
- `HISTORY_115`
- `HISTORY_2575`
- `HISTORY_1340`
- `NFL_988`
- `HISTORY_423`
- `HISTORY_1959`
- `NFL_29`
- `HISTORY_2867`
- `NFL_2191`
- `HISTORY_3754`
- `NFL_1021`
- `NFL_2269`
- `HISTORY_4060`
- `HISTORY_1773`
- `HISTORY_2757`
- `HISTORY_468`
- `HISTORY_10`
- `HISTORY_2151`
- `HISTORY_725`
- `NFL_858`
- `NFL_122`
- `HISTORY_591`
- `HISTORY_2948`
- `HISTORY_2829`
- `HISTORY_4034`
- `HISTORY_3717`
- `HISTORY_187`
- `HISTORY_1995`
- `NFL_1566`
- `HISTORY_685`
- `HISTORY_296`
- `HISTORY_1876`
- `HISTORY_2733`
- `HISTORY_325`
- `HISTORY_1898`
- `HISTORY_1948`
- `NFL_1838`
- `HISTORY_3993`
- `HISTORY_3366`
- `HISTORY_79`
- `NFL_2584`
- `HISTORY_3241`
- `HISTORY_1879`
- `HISTORY_2004`
- `HISTORY_4050`
- `NFL_2668`
- `HISTORY_3683`
- `HISTORY_836`
- `HISTORY_783`
- `HISTORY_2953`
- `HISTORY_1723`
- `NFL_378`
- `HISTORY_4137`
- `HISTORY_200`
- `HISTORY_502`
- `HISTORY_175`
- `HISTORY_3341`
- `HISTORY_2196`
- `HISTORY_9`
- `NFL_2385`
- `NFL_1879`
- `HISTORY_1298`
- `NFL_2272`
- `HISTORY_2170`
- `HISTORY_4080`
- `HISTORY_3669`
- `HISTORY_3647`
- `HISTORY_586`
- `NFL_1454`
- `HISTORY_2760`
- `HISTORY_1498`
- `HISTORY_1415`
- `HISTORY_2361`
- `NFL_915`
- `HISTORY_986`
- `HISTORY_1744`
- `HISTORY_1802`
- `HISTORY_3075`
- `HISTORY_2412`
- `NFL_832`
- `HISTORY_3435`
- `HISTORY_1306`
- `HISTORY_3089`
- `HISTORY_1002`
- `HISTORY_3949`
- `HISTORY_1445`
- `HISTORY_254`
- `HISTORY_991`
- `HISTORY_2530`
- `HISTORY_447`
- `HISTORY_2661`
- `HISTORY_1746`
- `HISTORY_347`
- `NFL_3009`
- `HISTORY_1814`
- `NFL_3126`
- `HISTORY_972`
- `NFL_2528`
- `HISTORY_2417`
- `NFL_1184`
- `HISTORY_59`
- `HISTORY_1811`
- `HISTORY_3115`
- `HISTORY_71`
- `HISTORY_1935`
- `HISTORY_2944`
- `HISTORY_1019`
- `HISTORY_887`
- `HISTORY_533`
- `NFL_3195`
- `HISTORY_3615`
- `HISTORY_4007`
- `HISTORY_2950`
- `NFL_1672`
- `HISTORY_2897`
- `HISTORY_1887`
- `HISTORY_2836`
- `NFL_3356`
- `HISTORY_1828`
- `HISTORY_3714`
- `NFL_2054`
- `HISTORY_2709`
- `NFL_1883`
- `NFL_2042`
- `HISTORY_2162`
- `NFL_2197`
- `NFL_2369`
- `HISTORY_2765`
- `HISTORY_2021`
- `NFL_1152`
- `HISTORY_2957`
- `HISTORY_1863`
- `HISTORY_2064`
- `HISTORY_4045`
- `HISTORY_3058`
- `NFL_153`
- `HISTORY_1074`
- `HISTORY_159`
- `HISTORY_455`
- `HISTORY_761`
- `HISTORY_1552`
- `NFL_1769`
- `NFL_880`
- `NFL_2234`
- `NFL_2995`
- `NFL_2823`
- `HISTORY_2179`
- `HISTORY_1891`
- `HISTORY_2474`
- `HISTORY_3062`
- `NFL_490`
- `HISTORY_1416`
- `HISTORY_415`
- `HISTORY_2609`
- `NFL_1618`
- `HISTORY_3749`
- `HISTORY_68`
- `HISTORY_4011`
- `NFL_2067`
- `NFL_610`
- `NFL_2568`
- `NFL_1689`
- `HISTORY_2044`
- `HISTORY_1844`
- `HISTORY_3992`
- `NFL_716`
- `NFL_825`
- `HISTORY_806`
- `NFL_194`
- `HISTORY_2970`
- `HISTORY_2878`
- `NFL_1652`
- `HISTORY_3804`
- `HISTORY_90`
- `NFL_16`
- `HISTORY_515`
- `HISTORY_1954`
- `HISTORY_2011`
- `HISTORY_2832`
- `HISTORY_228`
- `NFL_2907`
- `HISTORY_2752`
- `HISTORY_1352`
- `HISTORY_3244`
- `HISTORY_2941`
- `HISTORY_1227`
- `HISTORY_130`
- `HISTORY_3587`
- `HISTORY_69`
- `HISTORY_2676`
- `NFL_1768`
- `NFL_995`
- `HISTORY_809`
- `HISTORY_941`
- `HISTORY_3264`
- `NFL_1264`
- `HISTORY_1012`
- `HISTORY_1450`
- `HISTORY_1048`
- `NFL_719`
- `HISTORY_2762`
- `HISTORY_2086`
- `HISTORY_1259`
- `NFL_1240`
- `HISTORY_2234`
- `HISTORY_2102`
- `HISTORY_688`
- `NFL_2114`
- `HISTORY_1459`
- `HISTORY_1043`
- `HISTORY_3609`
- `NFL_1223`
- `HISTORY_417`
- `HISTORY_1884`
- `HISTORY_2390`
- `NFL_2671`
- `HISTORY_2298`
- `HISTORY_659`
- `HISTORY_459`
- `HISTORY_1542`
- `NFL_1914`
- `HISTORY_1258`
- `HISTORY_2164`
- `HISTORY_2777`
- `NFL_1304`
- `HISTORY_4049`
- `HISTORY_1423`
- `NFL_2994`
- `HISTORY_2814`
- `HISTORY_2187`
- `HISTORY_3280`
- `HISTORY_794`
- `NFL_3342`
- `HISTORY_2153`
- `HISTORY_1708`
- `NFL_1540`
- `HISTORY_92`
- `HISTORY_1907`
- `NFL_290`
- `NFL_1167`
- `HISTORY_2885`
- `HISTORY_2258`
- `HISTORY_1940`
- `HISTORY_2380`
- `NFL_1245`
- `HISTORY_3552`
- `HISTORY_534`
- `NFL_1193`
- `NFL_264`
- `NFL_275`
- `HISTORY_1042`
- `NFL_1829`
- `NFL_2571`
- `NFL_296`
- `NFL_199`
- `HISTORY_2434`
- `NFL_1486`
- `HISTORY_107`
- `HISTORY_371`
- `NFL_1361`
- `HISTORY_1212`
- `NFL_2036`
- `NFL_913`
- `HISTORY_2886`
- `HISTORY_2737`
- `HISTORY_487`
- `NFL_1516`
- `NFL_2894`
- `HISTORY_3692`
- `NFL_496`
- `HISTORY_2707`
- `HISTORY_655`
- `NFL_286`
- `HISTORY_13`
- `HISTORY_556`
- `NFL_962`
- `HISTORY_1517`
- `HISTORY_1130`
- `NFL_624`
- `NFL_2125`
- `NFL_1670`
- `HISTORY_512`
- `NFL_1515`
- `HISTORY_893`
- `HISTORY_1233`
- `HISTORY_3116`
- `HISTORY_544`
- `HISTORY_3807`
- `HISTORY_2088`
- `NFL_2601`
- `HISTORY_1952`
- `HISTORY_131`
- `HISTORY_3662`
- `HISTORY_883`
- `HISTORY_2949`
- `HISTORY_1965`
- `NFL_778`
- `HISTORY_2047`
- `HISTORY_4009`
- `HISTORY_520`
- `HISTORY_1748`
- `HISTORY_154`
- `NFL_493`
- `NFL_187`
- `HISTORY_1578`
- `NFL_1344`
- `NFL_3489`
- `NFL_246`
- `NFL_336`
- `NFL_3396`
- `NFL_816`
- `NFL_1390`
- `HISTORY_3363`
- `HISTORY_4002`
- `HISTORY_4141`
- `NFL_1378`
- `HISTORY_476`
- `NFL_477`
- `NFL_1471`
- `NFL_3420`
- `HISTORY_227`
- `HISTORY_3859`
- `NFL_715`
- `HISTORY_283`
- `HISTORY_1943`
- `HISTORY_1665`
- `HISTORY_1860`
- `NFL_2387`
- `HISTORY_3253`
- `HISTORY_2766`
- `HISTORY_671`
- `HISTORY_720`
- `HISTORY_3141`
- `HISTORY_1373`
- `HISTORY_2453`
- `HISTORY_3608`
- `HISTORY_343`
- `NFL_2918`
- `HISTORY_3866`
- `HISTORY_2818`
- `NFL_2330`
- `NFL_2636`
- `NFL_1553`
- `HISTORY_1082`
- `HISTORY_3900`
- `NFL_2202`
- `HISTORY_3404`
- `HISTORY_103`
- `NFL_2409`
- `NFL_1412`
- `HISTORY_2188`
- `NFL_3386`
- `NFL_1503`
- `NFL_1288`
- `NFL_2151`
- `NFL_1743`
- `HISTORY_2815`
- `HISTORY_2671`
- `HISTORY_1892`
- `NFL_613`
- `HISTORY_1356`
- `HISTORY_2363`
- `HISTORY_424`
- `HISTORY_3438`
- `HISTORY_148`
- `NFL_3290`
- `NFL_663`
- `HISTORY_732`
- `HISTORY_3092`
- `HISTORY_408`
- `NFL_3460`
- `HISTORY_2809`
- `HISTORY_530`
- `HISTORY_3588`
- `HISTORY_1853`
- `HISTORY_513`
- `HISTORY_918`
- `HISTORY_908`
- `HISTORY_2869`
- `HISTORY_1125`
- `HISTORY_796`
- `HISTORY_1601`
- `HISTORY_1250`
- `HISTORY_1092`
- `HISTORY_351`
- `HISTORY_2142`
- `NFL_2255`
- `HISTORY_3533`
- `HISTORY_3400`
- `HISTORY_2456`
- `HISTORY_3164`
- `HISTORY_2339`
- `NFL_2297`
- `HISTORY_3105`
- `NFL_1596`
- `NFL_2893`
- `HISTORY_539`
- `NFL_1332`
- `HISTORY_208`
- `NFL_350`
- `NFL_2645`
- `HISTORY_2921`
- `HISTORY_1167`
- `HISTORY_2892`
- `HISTORY_791`
- `NFL_3222`
- `NFL_1789`
- `NFL_180`
- `NFL_3594`
- `HISTORY_3143`
- `NFL_824`
- `NFL_2034`
